It's easier to see what's going wrong by looking at scala 2.12, since that's failing fewer tests. We instrument the message buffer nodes with:

  onType("org.apache.pekko.util.MessageBuffer$Node")
    .mixin(classOf[HasContext.Mixin])
    .advise(isConstructor, classOf[CaptureCurrentContextOnExit])
    .advise(method("apply"), classOf[InvokeWithCapturedContext])

which adds a context store to each message buffer node and instantiates it from the environment context on construction. This part works. What doesn't work is the .advise(method("apply"), classOf[InvokeWithCapturedContext]) step; presumably because the function call is inlined, we don't get a chance to apply our advice, so when the node is executed, the stored context is not propagated into the function call. This issue is even more prevalent in scala 2.13, but entirely missing from scala 3.

It may be the case that we can simply only support pekko 1.1 instrumentation in scala 3, although that's certainly not the ideal outcome...

So far, there are no great solutions. I have seen some similar issues with https://github.com/pjfanning/micrometer-pekko (itself a fork of Kamon from years ago). I decompiled a few Pekko classes for Scala 2.13 and the necessary methods for the instrumentation still seemed to be in the classes and in the call traces but still the instrumentation missed certain calls in practise.

Scala 2.13 users, in particular, and maybe Scala 2.12 users who really need Kamon can consider

• sticking with Pekko 1.0 - there are not massive changes in Pekko 1.1
• compiling Pekko 1.1 for themselves - as @hughsimpson mentioned, this requires -Dpekko.no.inline=true when you run the Pekko sbt build.

There are a few other projects out there that offer Pekko metrics that don't rely on instrumenting the classes at runtime but they require code changes to your code base to uptake.

Upstream changes in pekko are holding. I wonder if we should run some more automatic test on the nightlies to catch future changes earlier. I imagine pekko support is one of the bigger use cases of Kamon...

Upstream changes in pekko are holding. I wonder if we should run some more automatic test on the nightlies to catch future changes earlier. I imagine pekko support is one of the bigger use cases of Kamon...

No harm in testing against Pekko snapshots. The enabling of Scala 2 inlining was a big change and it would be rare for us to take on a large change like that.

Compiler might one day decide to inline something else we were depending on, and/or internals might change more. As you know I'd rather we had long term documented bindings, but I'm sure we both know how long that might take to materialise

Pushed a bumped pekko for pekko-http tests with failures

Working now with upstream changes